Great feedback, thanks! Will implement it later this week.

Edit: Sorry, it has been taking a bit longer, just started a new job. It's still on my mind though.

@all-contributors please add @maartenvanhooftds for code

@amit-sharma

I've put up a pull request to add @maartenvanhooftds! 🎉

@maartenvanhooftds Just realized that sometimes the test does not succeed. If you look at the CI build above, the result is around 0.61 which is lower than 0.7.
Can you double check the values that the code is producing? One fix may be to lower the threshold, but I think it may be useful to study the output and create a dataset where the result is always high.

Thanks for your patience @amit-sharma

If you ask me, two things are not going right:

1. Results have too low accuracy
2. The seed is not respected in the CI build.

To get some insights in 1), I've re-ran without seeds for 100 runs to get an insight in the accuracy distribution of CACM for val and test split. I've found that even when playing with higher signal (beta) on the dataloaders, I don't always get sufficient accuraccy, occassionaly there just are some outliers. So all in all the results are good, but some poor outliers.

I think in general, the nicest way to solve this is by seeding. Upper code passes on my machine 😄 So I must be missing something for setting the seed. Do you have an idea on either:
a) what seed I should've set that would be required for reproducibility in CI run?
OR b) how I can set up the same environment as CI locally, such that I can reproduce the CI results (which I can't now)?

thanks for looking into this, @maartenvanhooftds .
Looks like pytorch lightning has its own random seeds to be set too. Stack overflow

can you try adding seed_everything and deterministic=True?

Otherwise it may be a version mismatch in py3.11 (the test that is failing). There's not too much difference between the github CI env and a local installation, except the exact versions of the packages installed. you may want to check with py3.11 and the packages installed here in the log here

This PR is stale because it has been open for 60 days with no activity.

This PR was closed because it has been inactive for 7 days since being marked as stale.